Welcome back to pattern recognition.

So today we want to start looking into more details how to actually model the classifiers

and the different decision boundaries and we'll start with the example of logistic regression.

So, we're back with pattern recognition and today we want to start looking into logistic

regression.

Now, logistic regression is a discriminative model because it models the posterior probabilities

directly.

So we can essentially have a look at our posterior probability and let's say we have two classes

that are encoded as 0 and 1 and now we want to compute the probability of observing 0

given some observation x.

Now we know that we can apply Bayes' theorem and therefore we can rewrite the probability

in the following way.

So we have the probability of observing 0 at all and then we have the probability of

the observation given the particular class and we divide by the probability of observing

x at all.

Now we can essentially use this trick here where we do the marginalization of x so we

know that this is also expressible as the sum over the joint probabilities and here

we already replaced the joint probabilities with the respective composition with the priors

and the constraint probability and here you see that we essentially then replace it with

the sum in the denominator.

Now we can rearrange this a little bit by dividing the whole fraction with P of y equals

to 0 times P of x given y equals to 0 and if we do so we can get the following rearrangement

and you see that we end up with this double fraction and we see that we essentially have

the class conditional probabilities and the priors now all in the denominator.

We can further rearrange this a little bit and the idea now is that we want to extend

with the exponential and the logarithm so if we do so we get the following kind of relation.

We essentially have the same term and we take e to the power of a logarithm.

This would essentially cancel out again to be 1 so this doesn't change anything and

now we can do another trick.

We can use the logarithm in order to split the two and then you would see that we can

now rearrange into a fraction of the prior probabilities and we can rearrange into a

fraction of the class conditional probabilities.

Also we can observe that we are essentially using the Bayes rule here so we can also rearrange

it to the following equation where we then simply have the fraction of the posterior

probabilities so this would be equivalent in our case.

So you see that we found a way to rearrange our probability of y equals to zero given

x and we found a very interesting formulation here and we can actually see that this particular

shape is actually called the logistic function.

So the logistic function is given as 1 over 1 plus e to the power of minus capital F of

x and here capital F of x is actually the decision boundary.

So we can see that if we do the same for the other class where y equals to 1 we get the

following relationship so we have 1 minus p of y equals to zero given x and now we can

essentially plug in our previous definition and rearrange a little bit.

You can see now that we essentially can simplify this in another step and we get again a logistic

function but now with e to the power of F of x in the denominator.

So you see we essentially have two times the same formulation, the same function F of x

and on the one side it's minus F of x and on the other side it is F of x.

So this essentially means that F of x is describing how the probability is being assigned and

this is essentially nothing else than the decision boundary.

We will look into a little more detail in the next couple of slides.